Papers with modeling long sequences
Exploring the Limitations of Mamba in COPY and CoT Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Inference overhead of Transformers increases linearly with the sequence length, posing challenges for modeling long sequences. |
| Approach: | They analyze Mamba's expressive ability to perform COPY operations and Chain of Thought reasoning tasks using a defined sequence length. |
| Outcome: | The proposed model can perform COPY operations and Chain of Thought reasoning tasks with a constant size while reducing computational costs. |
MMDEND: Dendrite-Inspired Multi-Branch Multi-Compartment Parallel Spiking Neuron for Sequence Modeling (2025.acl-long)
Copied to clipboard
Kexin Wang, Yuhong Chou, Di Shang, Shijie Mei, Jiahong Zhang, Yanbin Huang, Man Yao, Bo Xu, Guoqi Li
| Challenge: | Vanilla spiking neurons are simplified from complex biological neurons with dendrites, soma, and synapses into single somatic compartments. |
| Approach: | They propose a multi-branch, multi-compartment parallel spiking dendritic neuron with a proportion-adjustable multi-branched structure that enables long-term temporal dependencies. |
| Outcome: | The proposed model achieves better long-sequence modeling capability with fewer parameters and lower energy consumption. |